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INSTITUTE OF EDUCATION SCIENCES 
GRANT APPLICATION REVIEW 

Under the Education Sciences Reform Act, activities of the Institute of Education Sciences (Institute) that 
are carried out through grants, contracts, or cooperative agreements, at a minimum, shall be awarded on 
a competitive basis and, when practicable, through a process of peer review. Further, the Director is 
required to establish a peer review system, involving highly qualified individuals with an in-depth 
knowledge of the subject to be investigated, for reviewing and evaluating all applications for grants and 
cooperative agreements that exceed $100,000, and for evaluating and assessing the products of research 
by all recipients of grants and cooperative agreements under this Act. 

See Appendix A for relevant sections of the legislation. 

Overview 

The Standards and Review Office is responsible for the Institute's scientific peer review process. 
Standards and Review staff work with a contractor that handles the logistics and manages the electronic 
system for grant application submission and review. The review process includes (a) pre-submission 
procedures that enable the Institute to plan for specific review sessions, (b) application processing 
procedures, (c) panel selection and appointment, (d) pre-review panel procedures, (e) panel meeting, 
and (0 post-review process. 



Pre-Submission Procedures 



Letters of Intent 

In the Request for Applications, the Institute encourages potential applicants to send a letter indicating 
their intent to submit an application. As described in the Request for Applications, the Letter of Intent is 
optional, is not binding, and does not enter into the review of subsequent applications. The information 
that it contains allows Institute staff to estimate the potential workload to plan the review. 

The Letter of Intent must be submitted electronically by the date listed at the beginning of the Request 
for Applications, using the instructions provided at the submission web site listed in the Request for 
Applications. 

The Letter of Intent includes a descriptive title, the goal that the application will address, and a brief 
description of the research project; the name, institutional affiliation, address, telephone number and e- 
mail address of the principal investigator(s); and the name and institutional affiliation of any key 
collaborators. The Letter of Intent indicates the duration of the proposed project and provides an 
estimated budget request by year, and a total budget request. 

Letters of Intent Receipt and Report 

The contractor provides a report on the Letters of Intent. This report details the number of Letters of 
Intent received for each competition and includes a .pdf file containing the letters received for each 
competition. 

Review of Letters of Intent 

Institute staff members review the Letters of Intent to estimate the numbers of applications within each 
competition and the distribution of applications across goals. Institute staff members also check the 
letters to identify particular applications that may require additional expertise on the review panel. 

The Letters of Intent are examined to identify potential applicants who may be considering the 
submission of applications that are likely to be non-responsive to the Request for Applications (e.g.. 
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individuals seeking support to implement a program in their school or individuals seeking scholarship 
funds to support their education). Program officers contact these individuals to clarify the Request for 
Applications and, when appropriate, to suggest alternative sources of funding. 

Processing Applications 



Submission of Applications 

Application forms and instructions for the electronic submission of applications are available at the web 
site identified in the Request for Applications approximately one month prior to the application due date. 
This date is announced in the Request for Applications. Information about the electronic submission 
procedures that must be followed and the software that will be required are available on the web site. 

Receipt of Applications 

Applicants must submit their application electronically by 8:00 p.m. Eastern time on the application 
receipt date announced in the Request for Applications, using the ED standard forms and the instructions 
provided on the application submission web site. 

Screening of Applications 

Applications are screened for compliance with the application rules (e.g., page length and formatting 
requirements, completion of all parts of the application), responsiveness to the Request for Applications, 
and identification of resubmissions. 

Compliance. Each application is screened for compliance with the application rules. Staff record 
when the application was submitted and verify that the required applicants components have been 
submitted and that each component meets format requirements and length restrictions. 

Responsiveness. Institute staff members screen each application for responsiveness to the 
Request for Applications. Applications are identified (a) that may have been submitted to the wrong 
competitions or to the wrong goal; (b) in which text in the abstract or narrative suggest that the 
applicant meant to submit to a particular goal but the application is currently listed under a different goal; 
or (c) that may be non-responsive. 

Staff members identify applications that may have been submitted to the wrong competition or to the 
wrong goal and send a report to the Senior Review Officer. If the Senior Review Officer concurs, then 
the applicant is contacted to verify the competition or goal to which the application has been submitted. 
On occasion, the Institute suggests that the application is more appropriate for a different competition or 
goal (e.g., in instances in which an application would be non-responsive for a particular competition but 
responsive for an alternate competition). In such cases, the Institute contacts the applicant to discuss 
the appropriate placement of the application. 

The review for responsiveness is not an evaluation of the quality of the proposal or of a component of 
the proposal. Non-responsive applications are those that do not meet the basic requirements with respect 
to restrictions on the target sample, type of intervention or assessment, and provision of detailed 
descriptions of research and data analytic procedures. Staff members identify applications that may be 
non-responsive and send reports to the Senior Review Officer. The Senior Review Officer reviews each 
identified application and makes a final determination of the responsiveness of the application. The 
Institute takes a conservative approach to responsiveness screening, and relatively few applications are 
judged to be non-responsive. 

Resubmission. Institute staff members screen each application for text in the abstract, narrative, 
or Appendix A that indicates that the applicant meant the application to be a resubmission but did not 
check the appropriate resubmission box on the application form. Applications classified as possible 
resubmissions that do not appear on the resubmission report for the competition are marked and a report 
is sent to the Senior Review Officer for review. The Senior Review Officer reviews each identified 
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application, and for those appearing to be resubmissions, the applicant is contacted. If the applicant 
confirms that the application is a resubmission, reviews from the previous year are sent along with the 
application to reviewers. 

Assignment of Applications to Pane! 

Applications are assigned to panel according to the match between the overall expertise of reviewers on 
each panel and the content and methodological approach proposed in each application. 

Assignment of Applications to Reviewers within Panels 

Initial Conflict of Interest Determination. Prior to assignment of applications. Institute staff review 
each application to identify possible conflicts of interest by comparing the individuals associated with 
each application (principal investigator, key personnel, consultants, advisory board members) against the 
individuals on each panel and the applicant institutions with the organizational affiliations of each panel 
member. In addition, any obvious professional collaboration between individuals associated with each 
application and panel members is noted. Further discussion of conflict of interest, including definitions of 
types of conflicts of interest, appears in the section below on Appointing Scientific Review Panels. 

Assignment of Applications. Applications are assigned to reviewers according to the match 
between the expertise of each reviewer and the content and methodological approach proposed in each 
application, with consideration given to the balance of applications across reviewers. 

Number and Type of Reviewers. Each application is assigned to at least 2 reviewers. Applications 
to conduct randomized trials submitted to Goal 3 (efficacy evaluations) or Goal 4 (effectiveness 
evaluations) have 3 reviewers, one of which is an experienced methodologist in the implementation and 
evaluation of randomized trials. Applications to conduct studies implementing single-subject 
methodologies are assigned at least one reviewer who is experienced in the implementation and analysis 
of single-subject studies. 

Selection and Appointment of Scientific Review Panels 

Overview of Pane! Composition and the Roles of Pane! Members 

Panels are composed of a Panel Chair, a Scientific Review Administrator, and panel members. The 
Deputy Director for Science appoints the Panel Chair from among the members of the panel. The 
Scientific Review Administrator is a non-voting member of the panel and is typically a Standards and 
Review staff member or contractor staff member. The Scientific Review Administrator understands the 
peer review process and is familiar with all of the procedural rules governing the process. The Panel 
Chair and Scientific Review Administrator work together to facilitate an orderly peer review meeting and 
ensure that all applications receive a fair review by the panel. 

Selection of Pane! Members 

The Institute identifies highly qualified potential reviewers who have the scientific expertise to evaluate 
the proposals on the criteria listed in the Request for Applications. Potential reviewers are identified 
primarily on the basis of quality of the research they have conducted and published in scientific peer- 
reviewed journals and the degree to which they are an in-depth expert in the research methods and 
subject matter that are relevant to the applications dealt with by any given panel. The diversity that 
individuals will bring to a panel on disciplinary, institutional, and other dimensions is also considered. 

The Institute uses 3 types of reviewers on its panels: principal panel members, rotating panel members, 
and ad hoc panel members. 

Principal pane! member. Principal panel members serve 1 to 3 year terms on a review panel or 
until the specific research grant competition assigned to the panel is discontinued. Only standing panels 
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(see below) have principal panel members. The Director reviews and approves all appointments of 
principal panel members. 

For a particular panel review meeting, the Institute may assign a principal panel member to serve on an 
alternate peer review panel as a principal panel member rather than on the principal panel member's 
regular panel. Such actions would occur, for example, if the scientist's expertise were needed on the 
alternate panel for that meeting. 

Rotating panel member. Rotating panel members are appointed to a particular panel (standing 
panel or single session panel) for one review session and serve as full members of the panel, receiving a 
full assignment of applications to review, attending the panel review meeting, and scoring all applications 
considered by the panel except those for which the individual has a conflict of interest. 

Ad hoc pane! member. Ad hoc panel members are appointed to a particular panel (standing panel 
or single session panel) for one review session and are typically assigned 2 to 4 applications to review. 

Ad hoc panel members participate in the full panel review meeting on those applications for which they 
served as a primary reviewer. They do not participate in and are not present for the discussion and 
scoring of any application for which they were not a primary reviewer. Their participation is typically via 
teleconference. Ad hoc reviewers are recruited when the number of applications received is greater than 
what a panel can efficiently handle or when particular expertise is needed for a specific application. 

Types of Panels 

The Institute has two types of Scientific Review Panels: standing panels and single session panels. In 
Appendix B are lists of the Institute's initial standing panels and single session panels for the fall 2005 
review session. 

Standing panels. Standing panels are scientific review panels to which panel members may be 
appointed for multiple, consecutive review sessions. 

Standing panels may be composed of principal panel members, rotating panel members, and ad hoc 
panel members. To promote continuity in standards and procedures across review sessions, at least 50 
percent and up to 100 percent of the members of a full panel are principal panel members. A typical full 
panel is limited to 15 to 20 members, in order to facilitate discussion of the applications. The remaining 
members of the full panel are rotating panel members or ad hoc panel members who serve on a panel 
for a specific grant review session. Generally, no more than 3 or 4 ad hoc panel members are used to 
supplement a panel. 

For each meeting of a peer review panel, the Deputy Director for Science shall appoint an individual to 
serve as chair of the panel. 

Single session panels. Single session panels are appointed for one review meeting. These panels 
may be composed of rotating panel members and ad hoc panel members. A typical full panel is limited 
to 15 to 20 members, in order to facilitate discussion of the applications. Generally, no more than 3 or 4 
ad hoc panel members are used to supplement a panel. 

Recruitment of Pane! Members 

The identification of potential panel members is a continuous process involving an annual call for 
nominations from the lES Board, consideration of lES report external reviewers, solicitation of 
suggestions from lES and Center experts, literature review, and networking with known experts in 
relevant fields. After receiving the Director's approval (for potential principal panel members), the 
Deputy Director for Science generally sends the initial letter (electronic mail) of invitation to all potential 
reviewers (including rotating and ad hoc). This contact is followed by phone calls and electronic mail to 
the potential reviewer from the Institute staff member responsible for the panel. 
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Appointment letters. Following the acceptance of panel members to standing panels, the Deputy 
Director for Science sends appointment letters to panel members indicating the term of their 
appointment. 

Conflicts of Interest 

The Institute is concerned with three types of conflicts of interest, which are described below: (a) 
personal financial interest, (b) personal or professional relationship, and (c) professional relationship with 
the applicant's institution. 

Persona! financial interest The Institute considers reviewers to be in conflict if they can benefit 
financially from the outcome of the review. An individual may not serve on a panel if the individual is 
part of an application that is being considered by the panel, regardless of the scope of the individual's 
role on the application (e.g., principal investigator, consultant, advisory board member). An individual 
may not serve on a panel if his or her spouse or partner, child, household member, or other relative with 
whom he or she has a close relationship has a financial interest in the outcome of the review. An 
individual may not serve on a panel if he or she has a financial interest in a for-profit organization that 
has an application being considered by the panel. 

For principal panel members, this means that the individual would not serve as a reviewer for that peer 
review session, but could serve on the panel for future sessions and could serve on an alternate panel for 
that particular peer review session. 

Personal or professional relationship. Reviewers may not serve as a primary reviewer for and must 
recuse themselves from participating in the panel discussion and scoring of any application from 
individuals with whom they have a close personal or professional relationship (e.g., collaborators on 
current projects, recent students, recent professors, personal friends). Judgments are determined on the 
basis of recency, frequency, and strength of the working relationship between the member and the 
applicant as reflected, for example, in co-authored publications. If the close personal relationship is one 
that constitutes a potential personal financial interest, then the individual may not serve on the panel as 
described in the previous section. 

Reviewers may not participate in the review (either as primary reviewers or as part of the panel 
discussion and scoring) of applications from individuals with whom they have professional differences 
that could reasonably be viewed as affecting the objectivity of their review. 

For conflicts of interest falling under this category, staff at the panel meeting records the reviewer's 
absence from the panel discussion. The reviewer does not receive access to the application or to the 
reviews of the application. 

Professional relationship with an applicant's institution. Reviewers may not serve as a primary 
reviewer for and must recuse themselves from participating in the panel discussion and scoring of any 
application submitted by other individuals from the institution or organization that employs them. 

For conflicts of interest falling under this category, staff at the panel meeting records the reviewer's 
absence from the panel discussion. The reviewer does not receive access to the application or to the 
reviews of the application. 



Pre-Pane! Meeting Process 



Contact with Pane! Members 

After individuals have agreed to participate on a panel, their names are given to the logistics contractor 
so that all necessary forms are completed (e.g.. Conflict of Interest, Consulting Agreement) prior to the 
release of applications to the panel and travel arrangements can be made in a timely fashion. 
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Introduction to Contractor. Following an individuars acceptance to participate in a panel review, 
the Institute staff member responsible for the panel sends an acknowledgement letter (electronic mail) to 
the panel member that includes an introduction to the logistics contractor. 

Initial Contact from Contractor. The logistics contractor sends an introductory letter (electronic 
mail) to each panel member. The contractor provides information on the electronic peer review system 
and sends all forms that need to be completed by the reviewers (e.g., Conflict of Interest). 



Instructions to Pane! Members 

There are two primary means for providing guidance to panel members prior to the actual panel review 
session - the pre-meeting orientation teleconference and the peer reviewer handbook. 

Pre-Meeting Orientation Teleconference. Approximately 6 to 7 weeks before the panel review 
session, a pre-meeting orientation teleconference is conducted with Institute staff, the logistics 
contractor, and the panel members. Generally 2 to 3 teleconference times are offered for each panel so 
that panel members may participate at a convenient time. The purpose of the pre-meeting orientation is 
to provide reviewers with an overview of the peer review process, identify key points in the relevant 
Requests for Applications, familiarize the reviewers with the scoring system, emphasize the need for 
looking over the applications as soon as they receive them to identify any conflicts of interest with 
assigned applications, and stress the importance of completing reviews by the due date so that we can 
triage the applications in time for reviewers to familiarize themselves with all of the applications to be 
discussed at the review session. 

Peer Review Handbook. Each panel member receives a Peer Review Flandbook. The Flandbook 
provides detailed information about the responsibilities of the panel members and the panel chair. 

Release of Applications to Reviewers 

About 6 to 7 weeks prior to the panel meeting, each reviewer receives hard copies of the applications 
that have been assigned to the reviewer. At this time, each reviewer is also given access to those 
applications that have been assigned to him or her through the electronic peer review system. In 
addition, if any of the assigned applications are resubmissions, the reviewer will receive the previous 
year's reviews for that application. 

Review Criteria 

All of the Institute's regular research competitions have a common goal structure that categorizes the 
types of research projects the Institute funds and the requirements for projects submitted under each 
goal. These requirements are in the Requests for Applications. The research standards of the Institute 
are, in part, embodied in the articulation of specific methodological requirements in the Requests for 
Applications. The standards are also enacted through the judgments of the scientific reviewers. As such, 
the standards of the Institute are reflected in the quality and type of scientists selected for the scientific 
review panels. 

The review criteria are described in each Request for Applications. With some exceptions (e.g., training 
grants, research and development centers), the criteria are the same. Reviewers are expected to assess 
the following aspects of an application in order to judge the likelihood that the proposed research will 
have a substantial impact on the pursuit of that goal: 

Significance. Does the applicant make a compelling case for the potential contribution of the 
project to the solution of an education problem? For development projects, does the applicant present a 
strong rationale - including a theoretical foundation and prior empirical evidence - justifying the 
development of the selected intervention or assessment? For cases in which the applicant proposes to 
evaluate an intervention, does the applicant present a strong rationale justifying the need to evaluate the 
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selected intervention (e.g., does prior evidence suggest that the intervention is likely to substantially 
improve student learning and achievement)? 

Research Plan. The primary methodological issue is whether the research plan is appropriate for 
answering the research questions or testing the proposed hypotheses. Reviewers are asked to consider 
whether the applicant presents (a) clear hypotheses or research questions; (b) clear descriptions of and 
strong rationales for the sample, the measures (including information on the reliability and validity of 
measures), data collection procedures, and research design; and (c) a detailed and well-justified data 
analysis plan. 

Personnel. Does the description of the personnel make it apparent that the principal investigator, 
project director, and other key personnel possess the training and experience and will commit sufficient 
time to competently implement the proposed research? 

Resources. Does the applicant have the facilities, equipment, supplies, and other resources 
required to support the proposed activities? Do the commitments of each partner show support for the 
implementation and success of the project? 

Review of Applications by the Primary Reviewers 

Typically, each panel member is assigned to be a primary reviewer for about 8 applications. The primary 
reviewers prepare written critiques of the applications to which they have been assigned, addressing the 
strengths and weaknesses of each application. Reviews are submitted electronically through the 
electronic peer review system. 

Scoring of Applications 

Primary reviewers assign a score for each review criterion (significance, research plan, personnel, and 
resources) and an overall quality score for each application. Reviewers are asked to score each 
application against an ideal application, as opposed to evaluating applications against the other 
applications submitted to a competition. 

Criteria Scores. Each criterion is rated on a 7-point scale from 1 (poor) to 7 (excellent), where 
intermediate values are treated as equal steps along the scale. 

Overall Scores. Each reviewer rates the overall quality of the application on a scale from 5.0 (poor) 
to 1.0 (outstanding). Reviewers are given the adjectival equivalents listed below to help anchor their 
scores: 



Overall Score Range 


Adjectival Equivalent 


1.0 to 1.5 


Outstanding 


1.6 to 2.0 


Excellent 


2.1 to 2.5 


Very Good 


2.6 to 3.0 


Good 


3.1 to 4.0 


Fair 


4.1 to 5.0 


Poor 



The overall score is not an average of the criterion scores or a mathematical derivative of the criterion 
scores. Each scientific reviewer is asked to thoughtfully consider each aspect of the application and make 
a determination of the overall quality of the proposal. 
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Triage of Applications 

To facilitate the discussion by the full panel of the most competitive applications, the Institute generally 
employs a system of triaging applications so that only the most competitive applications are discussed 
and scored by the full panel at the panel review meeting. Determination of the most competitive 
applications is based on the preliminary rank order of the average overall scores of the applications. 

Average Overall Score Report. Based on the overall scores assigned by primary reviewers, an 
average overall score for each application is calculated and is used to generate a preliminary rank order 
of applications. The Institute asks all reviewers to submit their scores and critiques of the applications 10 
business days prior to the panel meeting so that the determination of the applications to be considered at 
the panel meeting is completed with sufficient time for reviewers to become familiar with any of the 
applications for which they were not a primary reviewer. 

Triage Cut Point In general, the triage cut point is set so that the full panel discusses about 25 
applications. Triage cut points may vary across competitions when a panel reviews applications for more 
than one competition. 

Pane! Nominations of Applications to be Considered by the Full Panel. At the beginning of each 
panel meeting, panel members are given the opportunity to nominate for consideration by the full panel 
any application that they reviewed and judged to be of equal merit to the applications being considered 
by the full panel on the basis of their average overall score. This process is described in the section on 
the procedures of the panel meeting. 

Discrepant Scores. Primary reviewers may reach different judgments (i.e., scores) about the merit 
of a particular application. Institute staff members calculate the difference in overall scores between 
reviewers for each application and then the average of the differences between scores across 
applications. For applications receiving only 2 primary reviews, applications that have widely discrepant 
scores are identified and a third review by a panel member is obtained. All three reviewers' scores are 
then used to calculate the average overall scores of such applications. 

Applications to be Discussed by the Full Pane! 

After the triage has been determined, panel members are notified of the applications to be discussed and 
scored by the full panel. Notification is done as soon as possible to allow panel members sufficient time 
to become familiar with the applications that will be discussed by the full panel. 



Pane! Meeting 



Review of Procedures 

The evening before a panel review session, the Deputy Director for Science meets with the panel 
chairpersons to go over the panel meeting procedures and discuss any issues that have arisen during the 
application review period. 

Plenary Session 

Typically the panel sessions begin with a short plenary session during which members of all of the panels 
meet together. The Deputy Director for Science provides a brief overview of the competitions for the 
panel members. At the end of the Plenary Session, panels are dismissed to convene their separate 
review meetings. 

Pane! Orientation 

Each panel review session begins with a brief introduction and orientation. Panel members and all 
observers introduce themselves. The order of procedures for the panel and materials in the panel 
notebook are reviewed. 
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Nomination of Triaged Appiications for Discussion by the Fuii Panei 

The panel chair begins by asking panel members if anyone wishes to nominate for consideration by the 
full panel any applications that they reviewed that did not survive triage but in their opinion were of equal 
scientific and technical merit to applications that are being considered by the full panel. 

If a panel member chooses to nominate an application, he or she is given 2 minutes to present the case 
for including the application in the panel review. A vote of the full panel occurs and a simple majority 
vote determines whether the application is added to the panel's agenda. 

Order of Review 

Institute staff determines the order of review before the panel meeting. Typically, applications are first 
sorted by competition goal (e.g.. Development, Efficacy, Effectiveness), and then, within each goal, 
applications are sorted by competition (e.g., Reading and Writing Education Research, Reading and 
Writing Special Education Research). Within each goal and competition category, applications are 
randomly ordered to the extent possible (e.g., consideration is generally given to arranging the order so 
that the same reviewer is not a primary reviewer on consecutive applications). 

Primary reviewers. In the Order of Review, the primary reviewers are assigned to be Reviewer 1, 
Reviewer 2, or Reviewer 3. In the panel meeting. Reviewer 1 presents a short overview of the 
application in addition to his or her critique of the application. Reviewers 2 and 3 are asked not to repeat 
what Reviewer 1 has said, but to provide additional comments and to note where they might disagree 
with Reviewer 1. 

Note Taker. For each application, a panel member other than one of the primary reviewers for an 
application is assigned the task of being the note taker who provides a summary of the panel discussion. 

Review of Appiications 

The review of applications begins when the chair identifies the application to be reviewed and asks 
anyone who has a conflict of interest to leave the room. 

Scores of Primary Reviewers 

The scores of the Primary Reviewers are presented to the panel. 

Presentation of Appiication and Critiques by Primary Reviewers 

Reviewer 1 presents a brief description of the application and his or her critique of the application. 
Reviewer 2 then provides any additional comments about the application. If there is a third reviewer. 
Reviewer 3 also gives his or her critique. 

During these presentations the primary reviewers are invited to express any strong personal points of 
view or personal perspectives, positive or negative, with respect to the application. 

Panei Discussion 

After the presentations of the primary reviewers, the entire panel discusses the merits of the application. 
Any panelist intending to comment is invited to express any strong points of view or personal 
perspectives, positive or negative, with respect to the application being discussed. The chair and the 
Scientific Review Administrator are responsible for ensuring that the discussion focuses on the scientific 
and technical merit of the application. The chair is responsible for ensuring that the application is 
reviewed according to the criteria described in the Request for Applications. 

Comments on Budget 

After the discussion of the scientific and technical merit of the application, the primary reviewers may 
offer comments on the appropriateness of the budget. 
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Summary of Panel Discussion by Note Taker 

The note taker reads his or her summary of the panel discussion. The chair is responsible for ensuring 
that the summary reflects the issues that were raised by the panel. After hearing the summary, the chair 
asks panel members if they concur with the summary as presented or if any additions or corrections are 
necessary. 

Minority report. On occasion, 2 or more panel members disagree with the majority view of the 
application as presented by the note taker. In such instances, these reviewers may submit a comment 
reflecting their view of the application. 

Revisit Scores of Primary Reviewers 

After the panel discussion, the primary reviewers are given the opportunity to change any of their original 
scores. 

Revision of reviews. If a primary reviewer substantially changes his or her scores based on the 
panel discussion, the reviewer is asked to make any necessary revisions to his or her written review so 
that the scores are aligned with the review. 

Confidential Scoring by AH Panel Members 

After the primary reviewers have indicated any changes in their scores, all panel members, acting 
independently and individually, score the application. A scoring sheet is provided in the review panel 
notebook with applications listed in order of review. At the end of each day, the scoring sheets are 
collected. 

Funding Enthusiasm Scores 

After all of the applications have been discussed and scored, reviewers are asked to give each application 
a funding recommendation: (a) highly recommend funding, (b) recommend funding, or (c) do not 
recommend funding. 



Post-Review Session Procedures 

Summary Reports 

The Standards and Review Office provides two basic reports to the program officers after each panel 
meeting. 

Overaii score report. The Final Scientific Merit Report includes average criteria and average overall 
scores for each application. The averages are based on the scores provided by the full panel, excluding 
any members who were in conflict. 

Funding enthusiasm report. The Funding Enthusiasm Ratings Report is the average of the funding 
enthusiasm ratings provided by the full panel, excluding any members who were in conflict. 

Feedback to Applicants 

All applicants receive copies of the primary reviewers' critiques. For applications receiving full panel 
discussion, a summary of those comments about the application is also sent. These materials are sent to 
the principal investigator and to the applicant institution's authorized representative. 

Procedures for Responding to Applicant Inquiries 

Applicants who have questions regarding the review of their application are directed first to the cognizant 
program officer. If an applicant questions whether his or her proposal received appropriate treatment 
and is not satisfied with the program officer's feedback, the inquiry is directed to the Deputy Director for 
Science. The Deputy Director for Science reviews the process under which the proposal was handled to 
determine if there was anything irregular or inappropriate in the handling of the application, which is the 
only basis on which the results of a grant competition review can be put aside. A re-review of an 
application is the only remedy available for an application that the Deputy Director for Science judges to 
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have been mishandled. A re-review consists of a review of the same application, not a revised version, 
during the next scheduled panel review meeting for recurring competitions. The re-review will occur 
without panel access to the reviews from the contested review session. In instances where competitions 
occur infrequently, the application will be re-reviewed during the fiscal year when the next competition to 
which the application was submitted occurs. 
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Legislation Relevant to Scientific Peer Review 
20 U.S.C. 9501 et. seq. 

Education Sciences Reform Act of 2002 



SECTION 102. DEFINITIONS 

(18) Scientifically based research standards . 

(A) The term "scientifically based research standards" means research standards that— 

(i) apply rigorous, systematic, and objective methodology to obtain reliable and valid 
knowledge relevant to education activities and programs; and 

(ii) present findings and make claims that are appropriate to and supported by the 
methods that have been employed. 

(B) The term includes, appropriate to the research being conducted— 

(i) employing systematic, empirical methods that draw on observation or experiment; 

(ii) involving data analyses that are adequate to support the general findings; 

(iii) relying on measurements or observational methods that provide reliable data; 

(iv) making claims of causal relationships only in random assignment experiments or other 
designs (to the extent such designs substantially eliminate plausible competing explanations for 
the obtained results); 

(v) ensuring that studies and methods are presented in sufficient detail and clarity to allow 
for replication or, at a minimum, to offer the opportunity to build systematically on the findings of 
the research; 

(vi) obtaining acceptance by a peer-reviewed journal or approval by a panel of 
independent experts through a comparably rigorous, objective, and scientific review; and 

(vii) using research designs and methods appropriate to the research question posed. 

(19) Scientifically valid education evaluation . The term "scientifically valid education evaluation" 
means an evaluation that— 

(A) adheres to the highest possible standards of quality with respect to research design and 
statistical analysis; 

(B) provides an adequate description of the programs evaluated and, to the extent possible, 
examines the relationship between program implementation and program impacts; 

(C) provides an analysis of the results achieved by the program with respect to its projected 
effects; 

(D) employs experimental designs using random assignment, when feasible, and other 
research methodologies that allow for the strongest possible causal inferences when random 
assignment is not feasible; and 

(E) may study program implementation through a combination of scientifically valid and reliable 
methods. 

(20) Scientifically valid research . The term "scientifically valid research" includes applied research, 
basic research, and field-initiated research in which the rationale, design, and interpretation are soundly 
developed in accordance with scientifically based research standards. 
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PART A - THE INSTITUTE OF EDUCATION SCIENCES 
SECTION 114. OFFICE OF THE DIRECTOR 



(0 DUTIES . The duties of the Director shall include the following: 

(2) To ensure the methodology applied in conducting research, development, evaluation, and 
statistical analysis is consistent with the standards for such activities under this title. 

(7) To ensure that activities conducted or supported by the Institute are objective, secular, 
neutral, and nonideological and are free of partisan political influence and racial, cultural, gender, or 
regional bias. 

(g) EXPERT GUIDANCE AND ASSISTANCE . The Director may establish technical and scientific peer- 
review groups and scientific program advisory committees for research and evaluations that the 
Director determines are necessary to carry out the requirements of this title. The Director shall 
appoint such personnel, except that officers and employees of the United States shall comprise no 
more than Va of the members of any such group or committee and shall not receive additional 
compensation for their service as members of such a group or committee. The Director shall ensure 
that reviewers are highly qualified and capable to appraise education research and development 
projects. The Federal Advisory Committee Act (5 U.S.C. App.) shall not apply to a peer-review group 
or an advisory committee established under this subsection. 

SECTION 116. NATIONAL BOARD FOR EDUCATION SCIENCES. 



(b) DUTIES . The duties of the Board shall be the following: 

(3) To review and approve procedures for the technical and scientific peer review of the 
activities of the Institute. 

SECTION 120. COMPETITIVE AWARDS. 

Activities carried out under this Act through grants, contracts, or cooperative agreements, at a minimum, 
shall be awarded on a competitive basis and, when practicable, through a process of peer review. 



PART B - NATIONAL CENTER FOR EDUCATION RESEARCH 

SECTION 134. STANDARDS FOR CONDUCT AND EVALUATION OF RESEARCH. 

(a) GENERAL . In carrying out this part, the Research Commissioner shall— 

(1) ensure that all research conducted under the direction of the Research Center follows 
scientifically based research standards; 

(2) develop such other standards as may be necessary to govern the conduct and evaluation of 
all research, development, and wide dissemination activities carried out by the Research Center to 
assure that such activities meet the highest standards of professional excellence; 

(3) review the procedures utilized by the National Institutes of Flealth, the National Science 
Foundation, and other Federal departments or agencies engaged in research and development, and 
actively solicit recommendations from research organizations and members of the general public in 
the development of the standards described in paragraph (2); and 

(4) ensure that all research complies with Federal guidelines relating to research misconduct. 
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b) PEER REVIEW . 

(1) In General. The Director shall establish a peer review system, involving highly qualified 
individuals with an in-depth knowledge of the subject to be investigated, for reviewing and evaluating 
all applications for grants and cooperative agreements that exceed $100,000, and for evaluating and 
assessing the products of research by all recipients of grants and cooperative agreements under this 
Act. 

(2) Evaluation. The Research Commissioner shall 

(A) develop the procedures to be used in evaluating applications for research grants, 
cooperative agreements, and contracts, and specify the criteria and factors (including, as 
applicable, the use of longitudinal data linking test scores, enrollment, and graduation rates 
over time) which shall be considered in making such evaluations. 

PART E - NATIONAL CENTER FOR SPECIAL EDUCATION RESEARCH 
SECTION 175. ESTABLISHMENT 

(c) APPLICABILITY OF EDUCATION SCIENCES REFORM ACT OF 2002 . Parts A and F, and the 
standards for peer review of applications and for the conduct and evaluation of research under sections 
133(a) and 134, respectively, shall apply to the Secretary, the Director, and the Commissioner in carrying 
out this part. 
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